Vehicle Sensor Calibration System

ABSTRACT

A vehicle sensor calibration system can detect an SDV on a turntable surrounded by a plurality of fiducial targets, and rotate the turntable using a control mechanism to provide the sensor system of the SDV with a sensor view of the plurality of fiducial targets. The vehicle sensor calibration system can receive, over a communication link with the SDV, a data log corresponding to the sensor view from the sensor system of the SDV recorded as the SDV rotates on the turntable. Thereafter, the vehicle sensor calibration system can analyze the sensor data to determine a set of calibration parameters to calibrate the sensor system of the SDV.

BACKGROUND

For typical calibration of self-driving vehicle (SDV) or autonomousvehicle (AV) sensor systems, each sensor is individually calibratedprior to installation to obtain measurement accuracy prior toinstallation. Thereafter, the sensor is mounted to the vehicle andfurther positional calibrations may be conducted.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure herein is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings in which likereference numerals refer to similar elements, and in which:

FIG. 1 is a block diagram illustrating an example vehicle sensorcalibration system, as described herein;

FIG. 2 depicts an SDV being rotated on a turntable of a vehicle sensorcalibration facility;

FIG. 3 is a block diagram illustrating an example autonomous vehicle(AV) or self-driving vehicle (SDV) implementing a control system, asdescribed herein;

FIG. 4 is a flow chart describing an example method of implementing adata collection and calibration process for an SDV utilizing a vehiclesensor calibration system, according to examples described herein;

FIG. 5 is a block diagram illustrating a computer system upon whichexamples described herein may be implemented; and

FIG. 6 is a block diagram illustrating a computing system for an AV orSDV upon which examples described herein may be implemented.

DETAILED DESCRIPTION

Certain “full vehicle” calibration techniques of self-driving vehicle(SDV) sensor systems involve a complete accuracy and positioncalibration with all sensors in their final location on the SDV. Theadvantages of full vehicle calibration can include significant timesavings, negating any potential alteration of the sensors duringinstallation, and enabling recalibration after a period of time toquantify drift in the calibration parameters. However, current fullvehicle calibration techniques can still be quite precarious and laborintensive, typically involving manual handling of a calibration array ormanual setup of the calibration array to enable a human driver tomanually drive the SDV to expose the sensor systems to the calibrationarray. Outdoor implementations involving human drivers can result ininconsistencies between sensor data collections due to, for example,inconsistent lighting conditions or inconsistent drive pattern by ahuman driver. Furthermore, manual calibration corresponding to a crew oftechnicians physically moving a calibration array around an SDV toprovide full exposure of fiducials to the SDV's sensor system can alsoresult inconsistent paths, speed, and orientation, which also results ininconsistent data collection. Such inconsistencies can result in animperfect calibration of the SDV's sensor systems, which can impacton-board data processing and even safety.

Furthermore, an added challenge regarding full vehicle calibration isrelated to data collection. During data collection, all degrees offreedom must be observable in order to optimize depth, angles,distortions, scale, etc., due to their interrelatedness in thecalibration process. As an example, to measure depth accuracy, thecalibration may require that a target be observed at various depths inorder to model parameters to be fitted properly to the acquired sensordata. Given the wide range of parameters to optimize during calibration,the data collection can become a daunting task, where preciselychoreographed motions are required to position the sensors relative tofiducial targets.

To address the shortcomings of current calibration techniques, a vehiclesensor calibration system is described herein that can optimize certainsensor systems for self-driving vehicles (SDVs). The vehicle sensorcalibration system can include a repeatable and mechanized sequence ofpre-planned actions providing the sensor system of the SDV with acomplete sensor view of a standardized environment. In various examples,the vehicle sensor calibration system includes a moveable or rotatablesurface (e.g., a turntable) onto which SDVs can be positioned, and aplurality of fiducial targets disposed around the turntable to enablecalibration of SDV sensor systems. The vehicle sensor calibration systemcan comprise an indoor room into which SDVs can drive to positionthemselves on the turntable. In particular, an indoor facility for thevehicle sensor calibration system can further include a controlledlighting system to provide consistent conditions for the sensor datacollection. In certain aspects, the lighting conditions in the indoorfacility may be initially optimized by the vehicle sensor calibrationsystem in order to obtain most optimal signal to noise ratio on thevarious fiducial targets by the sensors. Once the SDV has beenpositioned on the turntable, a control mechanism of the turntable canrotate the turntable to enable the SDV to receive a full sensor view ofthe fiducial targets.

In some aspects, the vehicle sensor calibration system can operate tocalibrate a suite of sensors of the SDV (e.g., LIDAR, single camera,stereo camera, radar, proximity sensors, etc.). For example, the vehiclesensor calibration system can provide SDVs with an ideal calibrationenvironment in which each calibration session can result in consistentand normalized data collection from the SDV's sensor systems to runthrough mathematical models for each sensor system. In variousimplementations, the vehicle sensor calibration system can comprise anindoor facility into which an SDV drives onto the turntable surroundedby the fiducial markers or targets. According to some examples, thevehicle sensor calibration system can receive a data log correspondingto sensor data from the SDV that indicates the SDV entering the indoorfacility, driving onto the turntable, rotating on the turntable toexpose the sensor systems to the fiducial targets, and then driving offthe turntable to exit the facility.

The vehicle sensor calibration system may then analyze the sensor datain the data log for calibration. According to examples provided herein,the sensor data comprises at least LIDAR data from one or more LIDARsensors of the SDV. Additionally or alternatively, the sensor data cancomprise camera data, or stereoscopic camera data from one or morecameras of the SDV. Other examples of sensor data from the SDV caninclude radar data from one or more radar systems of the SDV, proximitysensor data (e.g., from infrared sensors), sonar data from one or moreacoustic sensors, and the like. For each sensor data set (e.g., for eachtype of sensor data and/or for each individual sensor of the SDV), thevehicle sensor calibration system can run a standardized mathematicalmodel on the sensor data set to determine a set of calibrationparameters in order to precisely calibrate the sensor. In certainaspects, the vehicle sensor calibration system implements a gradientdescent technique as a first-order optimization in order to firstcompare the initial uncalibrated or misaligned sensor setup to an idealor normalized sensor view represented by the mathematical models.

According to examples provided herein, the vehicle sensor calibrationsystem can determine a set of calibration parameters for the sensor,which can comprise a number of adjustments to the individual sensor inorder to match the sensor data from the sensor to the idealized sensorview represented by the mathematical model. Such adjustments cancomprise alignment adjustments to a centralized field of view (e.g.,re-centering a misaligned camera), angular range setting for a LIDARlaser strip, or directional adjustment to a LIDAR's photodetector array.Additionally or alternatively, the vehicle sensor calibration system canverify certain settings and/or provide settings specifications for anumber of controllable parameters of the sensor. For example, for LIDARsystems, the set of calibration parameters can comprise adjustments tolaser power settings, photodetector sensitivity settings, LIDAR stripsettings, and the like. As another example, for camera or stereo camerasystems, the vehicle sensor calibration system can verify and/or providesetting specifications for aperture settings, zoom settings, resolution,frame rate and/or shutter speed, color temperature settings, gain or ISOsettings, saturation and contrast settings, focus, and the like.

In certain implementations, the vehicle sensor calibration system canprovide the set as a calibration package to a human technician to enablemanual adjustment of the SDV's sensor systems to perfect thecalibration. In variations, the SDV can comprise a control system thatcan automatically make the adjustments based on the set of calibrationparameters in the calibration package. Accordingly, in suchimplementations, the vehicle sensor calibration system can transmit thecalibration package to the SDV to enable the SDV to auto-calibrate itssensor systems. Specifically, the set of calibration parameters canprovide the SDV with precise settings for each controllable aspect ofeach sensor system. Thus, the set of calibration parameters can causethe SDV to make precise adjustments to the LIDAR system(s), the cameraand/or stereo camera system(s), the radar system(s), proximity sensorsystem(s), etc. According to various examples, when adjustments are madeto the SDV's sensor systems, a final calibration may be performed usingthe vehicle sensor calibration facility to confirm that alignment andsettings are within precise tolerances.

Among other benefits, the examples described herein achieve a technicaleffect of optimizing calibration of SDV sensor systems for precision,accuracy, and also minimizing calibration time and labor. Furthermore,by standardizing SDV calibration with repeatable and consistent results,the vehicle sensor calibration system can make calibration andre-calibration of multiple SDV sensor systems routine and reliable. Withaccurate normalization of sensor calibration, SDVs can operate in normalpublic road conditions with optimal sensor settings to further optimizeon-board sensor data processing.

As used herein, a computing device refers to devices corresponding todesktop computers, cellular devices or smartphones, personal digitalassistants (PDAs), laptop computers, tablet devices, virtual reality(VR) and/or augmented reality (AR) devices, wearable computing devices,television (IP Television), etc., that can provide network connectivityand processing resources for communicating with the system over anetwork. A computing device can also correspond to custom hardware,in-vehicle devices, or on-board computers, etc. The computing device canalso operate a designated application configured to communicate with thenetwork service.

One or more examples described herein provide that methods, techniques,and actions performed by a computing device are performedprogrammatically, or as a computer-implemented method. Programmatically,as used herein, means through the use of code or computer-executableinstructions. These instructions can be stored in one or more memoryresources of the computing device. A programmatically performed step mayor may not be automatic.

One or more examples described herein can be implemented usingprogrammatic modules, engines, or components. A programmatic module,engine, or component can include a program, a sub-routine, a portion ofa program, or a software component or a hardware component capable ofperforming one or more stated tasks or functions. As used herein, amodule or component can exist on a hardware component independently ofother modules or components. Alternatively, a module or component can bea shared element or process of other modules, programs or machines.

Some examples described herein can generally require the use ofcomputing devices, including processing and memory resources. Forexample, one or more examples described herein may be implemented, inwhole or in part, on computing devices such as servers, desktopcomputers, cellular or smartphones, personal digital assistants (e.g.,PDAs), laptop computers, printers, digital picture frames, networkequipment (e.g., routers) and tablet devices. Memory, processing, andnetwork resources may all be used in connection with the establishment,use, or performance of any example described herein (including with theperformance of any method or with the implementation of any system).

Furthermore, one or more examples described herein may be implementedthrough the use of instructions that are executable by one or moreprocessors. These instructions may be carried on a computer-readablemedium. Machines shown or described with figures below provide examplesof processing resources and computer-readable mediums on whichinstructions for implementing examples disclosed herein can be carriedand/or executed. In particular, the numerous machines shown withexamples of the invention include processors and various forms of memoryfor holding data and instructions. Examples of computer-readable mediumsinclude permanent memory storage devices, such as hard drives onpersonal computers or servers. Other examples of computer storagemediums include portable storage units, such as CD or DVD units, flashmemory (such as those carried on smartphones, multifunctional devices ortablets), and magnetic memory. Computers, terminals, network enableddevices (e.g., mobile devices, such as cell phones) are all examples ofmachines and devices that utilize processors, memory, and instructionsstored on computer-readable mediums. Additionally, examples may beimplemented in the form of computer-programs, or a computer usablecarrier medium capable of carrying such a program.

Numerous examples are referenced herein in context of an autonomousvehicle (AV) or self-driving vehicle (SDV). An AV or SDV refers to anyvehicle which is operated in a state of automation with respect tosteering and propulsion. Different levels of autonomy may exist withrespect to AVs and SDVs. For example, some vehicles may enableautomation in limited scenarios, such as on highways, provided thatdrivers are present in the vehicle. More advanced AVs and SDVs can drivewithout any human assistance from within or external to the vehicle.

System Description

FIG. 1 is a block diagram illustrating an example vehicle sensorcalibration system, as described herein. The vehicle sensor calibrationsystem 100 can be implemented as one or more computing systems incombination with a number of automated mechanisms and fiducial targets106 to control or otherwise guide a self-driving vehicle (SDV) 180through a precise and repeatable procedure in order to provideconsistent and optimal calibration of the SDV's sensor systems. In someexamples, the automated mechanism can comprise a conveyer system thatlocks the SDV 180 to a track, pulling it past a set of pre-placedfiducial targets. According to examples provided herein, the automatedmechanism can comprise a turntable 105 onto which the SDV 180 ispositioned to be rotated in place.

In one example, the SDV 180 self-drives onto and off of the turntable105. For example, the vehicle sensor calibration system 100 can comprisean indoor controlled environment (such as shown in FIG. 2), and caninclude an entrance/exit location (or separate entrance and exitlocations) through which the SDV 180 can self-drive. In variations, thevehicle sensor calibration system 100 can include a combination conveyorsystem and turntable 105 to allow for automated control of the SDV 180in entering, positioning, and exiting the vehicle sensor calibrationsystem 100.

In some aspects, the vehicle sensor calibration system 100 can include avehicle sensing module 120 that can generate a trigger 122 for aturntable controller 115 when the SDV 180 is properly positioned on theturntable 105. In certain variations, the vehicle sensing module 120 cancomprise a number of sensors, such as pressure sensors or weightmeasurement sensors on the turntable itself, cameras positioned aroundthe turntable, and the like, and/or can comprise a communicationdevice(s) to exchange communications with the SDV 180. It is furthercontemplated that the vehicle sensing module 120 can trigger control ofany mechanized system that directs the SDV 180 through a calibrationsequence, such as a conveyor system, or a combination of conveyor systemand turntable with fiducial targets 106 disposed throughout.

In many examples, the vehicle sensing module 120 can detects the SDV 180on the turntable 105 and can generate the trigger 122 to cause theturntable controller 115 to generate pivot commands 117 executable topivot the turntable 105 with the SDV 180 thereon. As the turntable 105rotates, the sensor array 185 of the SDV 180 can be provided with asensor view of the fiducial targets 106, which can be placed at variouslocations and distances around the turntable 105. Additional fiducialtargets can be included for certain sensors (e.g., the LIDAR and stereocameras) for entering and exiting the indoor facility.

According to examples described herein, the fiducial targets 106 cancomprise specialized markers for calibrating the sensor array 185. Insome implementations, vehicle sensor calibration system 100 can includeseparate specialized fiducial markers for the LIDAR systems and thecamera systems. Additionally or alternatively, the fiducial targets 106can each comprise a distinct pattern (e.g., a checkerboard pattern) toprovide the LIDAR systems and camera systems with a standard dataacquisition environment in order to run a calibration operation that canfuse the acquired data for each sensor with common sensor viewsrepresented by mathematical models. Thus, the vehicle sensor calibrationsystem 100 can include an analysis engine 135 that can compare theacquired sensor data 184 from each of the sensors of the SDV's 180sensor array 185 with sensor models stored in a database 130.

Furthermore, the vehicle sensor calibration system 100 can include anenvironment control system 108 to provide consistent and optimizedenvironmental conditions for every calibration session. In one aspect,the environment control system 108 can optimize lighting conditions byreceiving lighting information 114 from a number of photodetectors 112included within the indoor facility. As an example, each fiducial target106 can include a photodetector 112, which can provide lightinginformation 114 to the environment control system 108, which can adjustbrightness in a targeted manner to ensure that the lighting conditionsare the same for each fiducial target 106 over each respective SDVcalibration procedure. Additionally, the environment control system 108can include climate and/or temperature control to provide optimaltemperature and/or humidity within the indoor facility through everycalibration session. Thus, the environment control system 108 canprovide precise environmental conditions—including lighting,temperature, humidity, etc.—to maximize signal to noise ratio for thesensor array 185 during calibration.

According to examples described herein, the vehicle sensor calibrationsystem 100 can include a vehicle interface 110 to receive sensor datalogs 182 from the SDVs 180 being calibrated. In some examples, thevehicle sensor calibration system 100 can access data logs 182corresponding to the SDV 180 traveling in known environments with knownlandmarks, which can also be utilized as fiducial markers. As describedbelow with respect to FIG. 3, SDVs may travel throughout a given regionby dynamically comparing raw sensor data from on-board sensors (e.g.,LIDAR and cameras) with prerecorded maps or sub-maps of the region. Forexample, the SDV can store sub-maps comprising processed LIDAR and/orstereo camera data of road segments that indicate static objects andlandmarks to enable the SDV to perform localization and determine poseat any given location in the region. Furthermore, the SDVs can accessand utilize current sub-maps (correlated to a current position of theSDV in the region) to compare with raw sensor data in order todynamically identify any potential hazards or objects of interest,dynamically run a probabilistic analysis for each detected object ofinterest, and determine whether action is required to avoid an incident.Such sensor data comparisons between the sub-maps and the raw sensordata may be stored in data logs 182 of the SDV 180. According toexamples provided herein, the vehicle sensor calibration system 100 canaccess such data logs 182, and in some examples, the analysis engine 135can run an analysis or test to determine whether a misalignment or otheranomaly exists on the SDV's 180 sensor array 185.

In certain implementations, the analysis engine 135 can identifyconfiguration problems with the SDV's 180 sensor array 185 just bycomparing the raw sensor data collected by the SDV 180, with the sub-mapdata stored on the SDV 180. Additionally or alternatively, the analysisengine 135 can collect one or more data logs 182 from the SDV 180 thatcorrespond to the SDV 180 acquiring data within the vehicle sensorcalibration system 100 facility. In one example, the analysis engine 135collects a single data log 182 that include sensor data 184 acquired inconnection with the SDV 180 at least one of entering the facility,rotating on the turntable 105, and/or exiting the facility.

As described herein, the vehicle sensor calibration system 100 caninclude a database 130 storing mathematical models for each sensor type(e.g., LIDAR, camera, radar, proximity, sonar) or for each individualsensor of each sensor type for individual sensor view comparison. Forexample, the SDV 180 can include multiple camera, stereo camera, andLIDAR systems each requiring individual calibration. For example,current SDVs in development include seven separate LIDAR sensors andover fifteen separate cameras, making manual calibration exceptionallylabor and time intensive. Thus, in various implementations, the vehiclesensor calibration system 100 can store normalized mathematical modelsfor each of the SDV's 180 sensor that represent a faultless or nearfaultless sensor view and configuration for that particular sensor. Inmany aspects, the database 130 can store LIDAR models 132 forcalibrating the LIDAR sensors of the SDV 180, and camera models 134 forcalibrating the cameras and/or stereoscopic cameras of the SDV 180.

In some aspects, the sensor data 184 can be streamed and analyzed by theanalysis engine 135 in real time. In other aspects, the SDV 180 cantransmit the data log(s) 182—corresponding to the sensor dataacquisition exposing the sensors to the fiducial targets 106 whilerotating on the turntable 105—to the vehicle interface 110 either via awireless connection or a plug-in interface after sensor dataacquisition. The analysis engine 135 can run the LIDAR sensor data froman individual LIDAR sensor of the SDV 180 through the respective LIDARmathematical model 133 for that LIDAR sensor. Furthermore, the analysisengine 135 can run the camera data from an individual camera of the SDV180 through the respective camera mathematical model 136 for thatparticular camera.

In further implementations, information from the vehicle motion system(e.g., the turntable 105 and/or conveyor system in alternativeimplementations) can be utilized by the vehicle sensor calibrationsystem 100 as part of the calibration parameters optimization. Forexample, position, angle, and/or location data from the motion systemcan provide the vehicle sensor calibration system 100 with additionalinformation to determine when certain sensors of the SDV 180 areactivated throughout the calibration session. Thus, in many examples,the individual sensors of the SDV 180 may be activated when pointing ata particular fiducial target 106 and then deactivated when not.Accordingly, the amount of data recorded may be reduced, which canfurther optimize the calibration process.

In various aspects, each mathematical model can determine aconfiguration state of its corresponding sensor with respect to anideal, common state. Thus, the configuration results 137 for eachanalyzed sensor of the SDV 180 can be generated by the analysis engine135 and submitted to a calibration engine 140. The configuration results137 can comprise the mathematical model results of running the sensordata 184—acquired by the sensor array 185 while SDV 180 is put throughthe predetermined calibration procedures (e.g., entering the facility,rotating on the turntable 105, and exiting the facility)—through therelevant mathematical models to provide a comparison between the currentsensor configuration of the sensor array 185 and an idealized commonsensor configuration represented by the models stored in the database130. For example, the mathematical models for the SDV's 180 camerasensors can take three-dimensional points in the sensor field andproject them onto image pixel coordinates. Additionally, themathematical models for the SDV's 180 LIDAR sensors can take raw angularrange measurements from the LIDAR sensors and project them ontothree-dimensional positions in the world. The configuration results 137can comprise a comparison between the ideal sensor views and/orconfigurations represented by the mathematical models and the actualuncalibrated data acquired by the sensors.

In certain aspects, the calibration engine 140 can process theconfiguration results 137 utilizing calibration logs 152 stored in adatabase 150. For example, the calibration logs 152 can provide thevarious configuration parameters of the individual LIDAR systems 153 andcamera systems 154 of the SDV's 180 sensor array 185. For the LIDARsystems 153, such parameters can include LIDAR laser alignment settings(e.g., strip alignment), photodetector alignment, laser power settings,photodetector sensitivity settings, output frequency settings, and thelike. For the camera systems 154, such parameters may include camerafield of view and alignment settings, aperture settings, zoom settings,resolution, frame rate and/or shutter speed, color temperature settings,gain or ISO settings, saturation and contrast settings, focus, and thelike.

According to certain examples, the calibration engine 140 can processthe configuration results 137 for the entire suite of LIDAR sensors andthe camera sensors (i.e., single lens and multiple lens systems) togenerate a calibration 142 that comprises the individual adjustments toalignment and controllable parameters of the various sensors of theSDV's 180 sensor array 185. In certain examples, the calibration 142 maybe utilized by technicians in order to make the manual adjustments tothe sensor array 185. Additionally or alternatively, the calibration 142can be transmitted to the SDV 180 via the vehicle interface 110 so thatthe SDV 180 can implement the calibration configurations. In certainimplementations, the SDV 180 can include a control system that canself-calibrate sensor systems like the LIDAR sensors and camera sensors.For example, the LIDAR strips for each LIDAR sensor can be aligned withone or more actuators operable by the control system. As anotherexample, the stereo camera systems can be motor-mounted and thusautomatically adjusted by the control system. Additionally oralternatively, the control system of the SDV 180 can adjust the variousconfigurable parameters of the LIDAR and camera systems describedherein. Accordingly, the calibration 142 can comprise individualadjustments to be made to each of the sensor systems of the sensor array185. In some aspects, the control system of the SDV 180 can utilize thecalibration 142 to calibrate the sensor array 185 automatically.

FIG. 2 depicts an SDV being rotated on a turntable of a vehicle sensorcalibration facility. In the below discussion with respect to FIG. 2,the vehicle sensor calibration system 100 as shown and described in FIG.1 may be included to, for example, operate certain components of thevehicle sensor calibration facility 200 as shown in FIG. 2. For example,the turntable 210 and/or the door(s) 212 of the facility 200 may beoperated via sensors (e.g., motion or pressure sensors) by the vehiclesensor calibration system 100 described with respect to FIG. 1.Furthermore, the vehicle sensor system 100 can be included as acomponent of the facility 200, or as a standalone system for use withany calibration environment. Referring to FIG. 2, an SDV 205 can includea sensor array (not shown) and can autonomously drive into the facility200 via an entrance location 214. In certain implementations, thefacility 200 can include one or more doors 212 that can close when theSDV 205 enters the facility 200 to ensure consistent lightingconditions. Still further, the temperature of the facility can becontrolled in order to provide constant calibration conditions formultiple SDVs.

In variations, the facility 200 can include a conveyor system thatguides or propels the SDV 205 into the facility 200. According toexamples described herein, the facility 200 can include a plurality offiducial targets 206 that can be sensed by the SDV's 205 sensor array asthe SDV 205 rotates on the turntable 210. The fiducial targets 206 canact as standardized references for measurement. Thus, the vehicle sensorcalibration system 100 can analyze the sensor data from the SDV 205using the fiducials 206 as calibration references to determine a set ofcalibration parameters for the SDV's 205 sensor array. Specifically, thefiducials 206 provide standard targets to enable the vehicle sensorcalibration system 100 to run the mathematical models for the SDV's 205individual sensor systems to determine the calibration adjustments forthose individual sensors.

FIG. 3 is a block diagram illustrating an example AV or SDV implementinga control system, as described herein. In an example of FIG. 3, acontrol system 320 can autonomously operate the SDV 300 in a givengeographic region for a variety of purposes, including transportservices (e.g., transport of humans, delivery services, etc.). Inexamples described, an autonomously driven vehicle can operate withouthuman control. For example, in the context of automobiles, anautonomously driven vehicle can steer, accelerate, shift, brake, andoperate lighting components. Some variations also recognize that anautonomous-capable vehicle can be operated either autonomously ormanually.

According to some examples, the control system 320 can utilize specificsensor resources in order to intelligently operate the vehicle 300 inmost common driving situations. For example, the control system 320 canoperate the vehicle 300 by autonomously operating the steering,accelerating, and braking systems of the vehicle 300 as the vehicleprogresses to a destination. The control system 320 can perform vehiclecontrol actions (e.g., braking, steering, accelerating) and routeplanning using sensor information, as well as other inputs (e.g.,transmissions from remote or local human operators, networkcommunication from other vehicles, etc.).

In an example of FIG. 3, the control system 320 includes a computer orprocessing system which operates to process sensor data that is obtainedon the vehicle 300 with respect to a road segment upon which the vehicle300 operates. The sensor data can be used to determine actions which areto be performed by the vehicle 300 in order for the vehicle 300 tocontinue on a route to a destination. In some variations, the controlsystem 320 can include other functionality, such as wirelesscommunication capabilities, to send and/or receive wirelesscommunications with one or more remote sources. In controlling thevehicle 300, the control system 320 can issue instructions and data,shown as commands 335, which programmatically control variouselectromechanical interfaces of the vehicle 300. The commands 335 canserve to control operational aspects of the vehicle 300, includingpropulsion, braking, steering, and auxiliary behavior (e.g., turninglights on).

The SDV 300 can be equipped with multiple types of sensors 301, 303which can combine to provide a computerized perception of the space andthe physical environment surrounding the vehicle 300. Likewise, thecontrol system 320 can operate within the SDV 300 to receive sensor data311 from the collection of sensors 301, 303, and to control variouselectromechanical interfaces for operating the vehicle 300 on roadways.

In more detail, the sensors 301, 303 operate to collectively obtain acomplete sensor view for the vehicle 300, and further to obtainsituational information proximate to the vehicle 300, including anypotential hazards proximate to the vehicle 300. By way of example, thesensors 301, 303 can include multiple sets of cameras sensors 301 (videocameras, stereoscopic pairs of cameras or depth perception cameras, longrange cameras), remote detection sensors 303 such as provided by radaror LIDAR, proximity or touch sensors, and/or sonar sensors (not shown).

Each of the sensors 301, 303 can communicate with the control system 320utilizing a corresponding sensor interface 310, 312. Each of the sensorinterfaces 310, 312 can include, for example, hardware and/or otherlogical components which are coupled or otherwise provided with therespective sensor. For example, the sensors 301, 303 can include a videocamera and/or stereoscopic camera set which continually generates imagedata of the physical environment of the vehicle 300. As an addition oralternative, the sensor interfaces 310, 312 can include dedicatedprocessing resources, such as provided with field programmable gatearrays (FPGAs) which can, for example, receive and/or process raw imagedata from the camera sensor.

In some examples, the sensor interfaces 310, 312 can include logic, suchas provided with hardware and/or programming, to process sensor data 309from a respective sensor 301, 303. The processed sensor data 309 can beoutputted as sensor data 311. As an addition or variation, the controlsystem 320 can also include logic for processing raw or pre-processedsensor data 309.

According to one implementation, the vehicle interface subsystem 350 caninclude or control multiple interfaces to control mechanisms of thevehicle 300. The vehicle interface subsystem 350 can include apropulsion interface 352 to electrically (or through programming)control a propulsion component (e.g., an accelerator actuator), asteering interface 354 for a steering mechanism, a braking interface 356for a braking component, and a lighting/auxiliary interface 358 forexterior lights of the vehicle 300.

The controller(s) 340 can generate control signals 349 in response toreceiving the commands 335 for one or more of the vehicle interfaces352, 354, 356, 358. The controllers 340 can use the commands 335 asinput to control propulsion, steering, braking, and/or other vehiclebehavior while the SDV 300 follows a current route. Thus, while thevehicle 300 actively drives along the current route, the controller(s)340 can continuously adjust and alter the movement of the vehicle 300 inresponse to receiving a corresponding set of commands 335 from thecontrol system 320. Absent events or conditions which affect theconfidence of the vehicle 320 in safely progressing along the route, thecontrol system 320 can generate additional commands 335 from which thecontroller(s) 340 can generate various vehicle control signals 349 forthe different interfaces of the vehicle interface subsystem 350.

According to examples, the commands 335 can specify actions to beperformed by the vehicle 300. The actions can correlate to one ormultiple vehicle control mechanisms (e.g., steering mechanism, brakes,etc.). The commands 335 can specify the actions, along with attributessuch as magnitude, duration, directionality, or other operationalcharacteristics of the vehicle 300. By way of example, the commands 335generated from the control system 320 can specify a relative location ofa road segment which the SDV 300 is to occupy while in motion (e.g.,changing lanes, moving into a center divider or towards a shoulder,turning the vehicle, etc.). As other examples, the commands 335 canspecify a speed, a change in acceleration (or deceleration) from brakingor accelerating, a turning action, or a state change of exteriorlighting or other components. The controllers 340 can translate thecommands 335 into control signals 349 for a corresponding interface ofthe vehicle interface subsystem 350. The control signals 349 can takethe form of electrical signals which correlate to the specified vehicleaction by virtue of electrical characteristics that have attributes formagnitude, duration, frequency or pulse, or other electricalcharacteristics.

In an example of FIG. 3, the control system 320 can include a routeplanner 322, event logic 324, and vehicle control logic 328. The vehiclecontrol logic 328 can convert alerts of event logic 324 (“event alert329”) into commands 335 that specify a set of vehicle actions.Furthermore, in operating the acceleration, braking, and steeringsystems of the SDV 300, the control system 320 can include a database370 that stores previously recorded and processed sub-maps 372 of agiven road segment or region.

As described herein, the sub-maps 372 can comprise processed ornormalized surface data that enables the control system 320 to comparewith the sensor data 311 in order to identify any potential hazards orobjects of interest.

Additionally, the route planner 322 can select one or more routesegments 326 that collectively form a path of travel for the SDV 300when the vehicle 300 is on a current trip (e.g., servicing a pick-uprequest). In one implementation, the route planner 322 can specify routesegments 326 of a planned vehicle path which defines turn by turndirections for the vehicle 300 at any given time during the trip. Theroute planner 322 may utilize the sensor interface 312 to receive GPSinformation as sensor data 311. The vehicle control 328 can processroute updates from the route planner 322 as commands 335 to progressalong a path or route using default driving rules and actions (e.g.,moderate steering and speed).

In certain implementations, the event logic 324 can trigger low levelresponses to detected events. A detected event can correspond to aroadway condition, obstacle, or object of interest which, when detected,poses a potential hazard or threat of collision to the vehicle 300. Byway of example, a detected event can include an object in the roadsegment, heavy traffic ahead, and/or wetness or other environmentalconditions on the road segment. The event logic 324 can use sensor data311 from cameras, LIDAR, radar, sonar, or various other image or sensorcomponent sets in order to detect the presence of such events asdescribed. For example, the event logic 324 can detect potholes, debris,objects projected to be on a collision trajectory, and the like. Thus,the event logic 324 can detect events which enable the control system320 to make evasive actions or plan for any potential hazards.

When events are detected, the event logic 324 can signal an event alert329 that can classify the event and indicates the type of avoidanceaction to be performed. Additionally, the control system 320 candetermine whether an event corresponds to a potential incident with ahuman driven vehicle, a pedestrian, or other human entity external tothe SDV 300. In turn, the vehicle control 328 can determine a low levelresponse based on a score or classification of the event. Such responsecan correspond to an event avoidance action 323, or an action that thevehicle 300 can perform to maneuver the vehicle 300 based on thedetected event and its score or classification. By way of example, thevehicle response can include a slight or sharp vehicle maneuver foravoidance using a steering control mechanism and/or braking component.The event avoidance action 323 can be signaled through the commands 335for controllers 340 of the vehicle interface subsystem 350.

When an anticipated dynamic object of a particular class does in factmove into position of likely collision or interference, some examplesprovide that event logic 324 can signal the event alert 329 to cause thevehicle control 328 to generate commands 335 that correspond to an eventavoidance action 323. For example, in the event of an incident in thepath of the vehicle 300, the event logic 324 can signal the event alert329 to avoid a collision. The event alert 329 can indicate (i) aclassification of the event (e.g., “serious” and/or “immediate”), (ii)information about the event, such as the type of object that generatedthe event alert 329, and/or information indicating a type of action thevehicle 300 should take (e.g., location of object relative to path ofvehicle, size or type of object, etc.).

According to examples described herein, SDV 300 can include acommunications module 360 to communicate with, among other things, thevehicle sensor calibration system 380, such as the vehicle sensorcalibration system 100 described with respect to FIG. 1. Paramount toefficient and safe operation of the SDV 300 on public roads is theaccurate calibration of the sensor systems 301, 303. As describedherein, the SDV 300 can record sensor data while operating through theprocedures of the vehicle sensor calibration system 380, and store theacquired sensor data in a data log 374. The sensor data in the data log374 can comprise at least LIDAR and camera and/or stereo-camera datafrom the various sensor systems of the SDV 300 as the SDV 300 enters thevehicle sensor calibration facility 200, rotates on the turntable 210,and exits the facility 200. Furthermore, the sensor data in the data log374 can include sensor views of the fiducial markers distributed in setpositions within the facility 200.

In certain implementations, the SDV 300 can transmit the data log 374 tothe vehicle sensor calibration system 380 via the communications module360. As described herein, the vehicle sensor calibration system 380 cananalyze the sensor data in the data log 374 using mathematical models inorder to generate a sensor calibration package 382 that enablestechnicians to manually calibrate the SDV's 300 sensors, or enables theSDV 300 to self-calibrate the sensor systems. Accordingly, in someaspects, the vehicle sensor calibration system 380 can transmit thesensor calibration package 382 to the SDV 300. The control system 320can utilize the sensor calibration package 382 to generate a set ofadjustment commands 384 for each of the uncalibrated or misalignedsensors. The set of adjustment commands 384 can calibrate, for example,laser alignment, laser power, photodetector alignment, photodetectorsensitivity, camera alignment or field of view settings, and the like.

Methodology

FIG. 4 is a flow chart describing an example method of implementing adata collection and calibration process for an SDV utilizing a vehiclesensor calibration system, according to examples described herein. Inthe below description of FIG. 4, reference may be made to referencecharacters representing like features as shown and described withrespect to FIGS. 1-3. Furthermore, the method described in connectionwith FIG. 4 may be performed by an example vehicle sensor calibrationsystem 100 as described with respect to FIG. 1. Referring to FIG. 4, thevehicle sensor calibration system 100 can rotate a turntable 105 havinga SDV 180 positioned thereon (400). According to examples, rotation ofthe turntable 105 can expose the sensor array 185 of the SDV 180 tovarious fiducial targets 106 disposed around the turntable 105 (402).Furthermore, in some aspects, the execution of the turntable 105 can beperformed in response to detecting the SDV 180 entering the vehiclesensor calibration facility 200 and being positioned atop the turntable105 (404).

According to various aspects, the vehicle sensor calibration system 100can receive sensor data 184 from the SDV 180 (410). In one example, thesensor data 184 is streamed to the vehicle sensor calibration system 100for real time analysis (412). In variations, the vehicle sensorcalibration system 100 can access a data log 182 corresponding to thesensor data scans of the fiducial targets 106 while in the facility 200(414). The vehicle sensor calibration system 100 can then run therelevant mathematical models on the sensor data 184 (415). For example,the vehicle sensor calibration system 100 can store mathematical modelsrepresenting ideal sensor configurations for each of the SDV's 180 LIDARsensors systems and camera and/or stereo-camera systems. Thus, thevehicle sensor calibration system 100 can run the respectivemathematical models on the sensor data acquired by each of the LIDARsystems (417), and each of the camera systems (419). In certain aspects,the mathematical models implement gradient descent alignment tocalibrate each of the SDV's 180 sensor systems.

Additionally, the vehicle sensor calibration system 100 can storemathematical models for other sensor systems, such as radar, sonar, andinfrared sensor systems. Thus, the vehicle sensor calibration system 100can further run respective mathematical models for the SDV's 180 radarsystems (416), sonar systems (418), and the various other sensor systemsof the SDV 180 (421).

For each of the sensors of the SDV 180, the vehicle sensor configurationsystem 100 can generate configuration results 137 that provides acomparison between the current configuration of the sensor and an idealcommon configuration represented by the mathematical model for thatsensor (420). For example, the configuration results 137 for aparticular LIDAR system can indicate a misalignment of certainindividual lasers, the entire laser strip, individual photodetectors,and/or the photodetector array. Furthermore, in some aspects, theconfiguration results 137 can further indicate incorrect or uncalibratedsettings of certain configurable parameters of the LIDAR system, such aspower settings, frequency settings, and the like.

In certain implementations, based on the configuration results 137, thevehicle sensor calibration system 100 can determine a set of calibrationparameters for each of the sensor systems of the SDV's 180 sensor array185 (425). As described herein, the calibration parameters can includealignment parameters indicating adjustments required to calibrate thealignment of the sensor (e.g., a camera's field of view), or certainaspects of the sensor (e.g., lasers of a LIDAR system) (427).Additionally or alternatively, the calibration parameters can comprisesettings adjustments to the various controllable aspects of the sensor,as described herein (429). Thus, calibration of the suite of sensors inthe sensor array 185 using the various calibration parameters determinedby the vehicle sensor calibration system 100 can provide the SDV 180with an accurate ground truth for dynamic real-world analysis duringoperation in traffic and on public roads and highways. As describedherein, the configuration results 137 and/or the calibration parametersbased on the configuration results 137 can be utilized by human orrobotic technicians in order to manually calibrate each of the sensorsof the sensor array 185. As an addition or alternative, the calibrationparameters for each of the sensor systems can be packaged in acalibration package 142, and transmitted to the SDV 180 for automaticcalibration (430).

Hardware Diagrams

FIG. 5 is a block diagram that illustrates a computer system upon whichexamples described herein may be implemented. A computer system 500 canbe implemented on, for example, a server or combination of servers. Forexample, the computer system 500 may be implemented as part of a networkservice for providing transportation services. In the context of FIG. 1,the vehicle sensor calibration system 100 may be implemented using acomputer system 500 such as described by FIG. 5. The vehicle sensorcalibration system 100 may also be implemented using a combination ofmultiple computer systems as described in connection with FIG. 5.

In one implementation, the computer system 500 includes processingresources 510, a main memory 520, a read-only memory (ROM) 530, astorage device 540, and a communication interface 550. The computersystem 500 includes at least one processor 510 for processinginformation stored in the main memory 520, such as provided by a randomaccess memory (RAM) or other dynamic storage device, for storinginformation and instructions which are executable by the processor 510.The main memory 520 also may be used for storing temporary variables orother intermediate information during execution of instructions to beexecuted by the processor 510. The computer system 500 may also includethe ROM 530 or other static storage device for storing staticinformation and instructions for the processor 510. A storage device540, such as a magnetic disk or optical disk, is provided for storinginformation and instructions.

The communication interface 550 enables the computer system 500 tocommunicate with one or more networks 580 (e.g., cellular network)through use of the network link (wireless or wired). Using the networklink, the computer system 500 can communicate with one or more computingdevices, one or more servers or an SDV 180. In accordance with examples,the computer system 500 can receive a data log 582 corresponding to theSDV 180 being run through a pre-planned sequence of motions (e.g.,rotating on a turntable 105) that expose the sensors of the SDV 180 to anumber of positioned fiducial markers 106. The executable instructionsstored in the memory 520 can include sensor data analysis instructions524, which the processor 510 can execute to analyze the sensor data inthe data log 582 for calibration. Execution of the sensor data analysisinstructions 524 can comprise running the sensor data in the data log582 through individual mathematical models for each sensor type (e.g.,LIDAR or camera), or for each individual sensor of each sensor type(e.g., main LIDAR, rear left corner stereo-camera, etc.). Theconfiguration results for each data set corresponding to each sensor canbe analyzed by the processor 510 to generate a calibration package 554.As described herein the calibration package 554 can be utilized formanual calibration or can be transmitted to the SDV 180 via thecommunication interface 550.

The processor 510 is configured with software and/or other logic toperform one or more processes, steps and other functions described withimplementations, such as described in connection with FIGS. 1 through 4,and elsewhere in the present application.

FIG. 6 is a block diagram illustrating a computer system upon whichexample SDV processing systems described herein may be implemented. Thecomputer system 600 can be implemented using one or more processors 604,and one or more memory resources 606. In the context of FIG. 3, thecontrol system 320 can be implemented using one or more components ofthe computer system 600 shown in FIG. 6.

According to some examples, the computer system 600 may be implementedwithin an autonomous vehicle or self-driving vehicle with software andhardware resources such as described with examples of FIG. 3. In anexample shown, the computer system 600 can be distributed spatially intovarious regions of the SDV, with various aspects integrated with othercomponents of the SDV itself. For example, the processors 604 and/ormemory resources 606 can be provided in the trunk of the SDV. Thevarious processing resources 604 of the computer system 600 can alsoexecute control instructions 612 using microprocessors or integratedcircuits. In some examples, the control instructions 612 can be executedby the processing resources 604 or using field-programmable gate arrays(FPGAs).

In an example of FIG. 6, the computer system 600 can include acommunication interface 650 that can enable communications over one ormore networks 660 with a vehicle sensor configuration system, such asthose examples described with respect to FIG. 1. In one implementation,the communication interface 650 can also provide a data bus or otherlocal links to electro-mechanical interfaces of the vehicle, such aswireless or wired links to and from the SDV control system 320, and canprovide a network link to a transport system or calibration system overone or more networks 660.

The memory resources 606 can include, for example, main memory, aread-only memory (ROM), storage device, and cache resources. The mainmemory of memory resources 606 can include random access memory (RAM) orother dynamic storage device, for storing information and instructionswhich are executable by the processors 604. The processors 604 canexecute instructions for processing information stored with the mainmemory of the memory resources 606. The main memory 606 can also storetemporary variables or other intermediate information which can be usedduring execution of instructions by one or more of the processors 604.The memory resources 606 can also include ROM or other static storagedevice for storing static information and instructions for one or moreof the processors 604. The memory resources 606 can also include otherforms of memory devices and components, such as a magnetic disk oroptical disk, for purpose of storing information and instructions foruse by one or more of the processors 604.

According to some examples, the memory 606 may store a set of softwareinstructions including, for example, control instructions 612. Thecontrol instructions 612 may be executed by one or more of theprocessors 604 in order to implement functionality such as describedwith respect to the SDVs herein. The computer system 600 can communicatea data log 654—corresponding to sensor data acquisition during acalibration procedure—to the vehicle sensor calibration system 100 overthe network 660. As described herein, the vehicle sensor calibrationsystem 100 can analyze the data log 654 using mathematical models togenerate a calibration package 662. As provided herein, thecommunication interface 650 can receive the calibration package 662,which the processor 604 can utilize to generate adjustment commands 634in order to calibrate the sensor array 630.

Furthermore, the processors 604 operate the acceleration, braking, andsteering systems 620 along a current route, either determined locallyvia a mapping resource, or via information from a remote source. Thus,in executing the control instructions 612, the processor(s) 604 canreceive sensor data 632 from the calibrated sensor array 630 todynamically generate control commands 605 to autonomously operate theacceleration, braking, and steering systems 620 of the SDV 180 throughroad traffic on roads and highways.

While examples of FIGS. 5 and 6 provide for computing systems forimplementing aspects described, some or all of the functionalitydescribed with respect to one computing system of FIGS. 5 and 6 may beperformed by other computing systems described with respect to FIGS. 5and 6.

It is contemplated for examples described herein to extend to individualelements and concepts described herein, independently of other concepts,ideas or systems, as well as for examples to include combinations ofelements recited anywhere in this application. Although examples aredescribed in detail herein with reference to the accompanying drawings,it is to be understood that the concepts are not limited to thoseprecise examples. As such, many modifications and variations will beapparent to practitioners skilled in this art. Accordingly, it isintended that the scope of the concepts be defined by the followingclaims and their equivalents. Furthermore, it is contemplated that aparticular feature described either individually or as part of anexample can be combined with other individually described features, orparts of other examples, even if the other features and examples make nomentioned of the particular feature. Thus, the absence of describingcombinations should not preclude claiming rights to such combinations.

1.-20. (canceled)
 21. A computing system comprising: one or moreprocessors; and one or more memory resources storing instructions that,when executed by the one or more processors, cause the one or moreprocessors to perform operations comprising: receiving sensor datarecorded by a sensor system comprising a sensor, the sensor positionedon a moveable surface, wherein the sensor data is associated with atleast one of a plurality of targets and the sensor data is recordedwhile the sensor system is positioned on the moveable surface;determining one or more calibration parameters to calibrate the sensorsystem based at least in part on the sensor data; and communicating datafor calibration of the sensor system based at least in part on thesensor data recorded by the sensor system and the one or morecalibration parameters.
 22. The computing system of claim 21, whereinthe sensor system is located on a vehicle and wherein the computingsystem is remote from the vehicle.
 23. The computing system of claim 21,wherein communicating the data for calibration of the sensor systembased at least in part on the sensor data recorded by the sensor systemand the one or more calibration parameters comprises: communicating acalibration package for automatic calibration of the sensor.
 24. Thecomputing system of claim 23, wherein the automatic calibration of thesensor is performed onboard a vehicle.
 25. The computing system of claim21, wherein communicating the data for calibration of the sensor systembased at least in part on the sensor data recorded by the sensor systemand the one or more calibration parameters comprises: communicating acalibration package for manual calibration of the sensor.
 26. Thecomputing system of claim 21, wherein the moveable surface is locatedwithin an indoor space.
 27. The computing system of claim 21, whereinthe operations further comprise: adjusting one or more environmentalconditions of the indoor space.
 28. The computing system of claim 27,wherein adjusting the one or more environmental conditions of the indoorspace comprises: communicating with another computing system to adjustthe one or more environmental conditions of the indoor space.
 29. Thecomputing system of claim 28, wherein the one or more environmentparameters conditions comprise (i) a lighting, (ii) a humidity of theindoor space, or (iii) a temperature.
 30. The computing system of claim21, wherein the calibration parameters are associated with a particulartype of sensor.
 31. The computing system of claim 29, wherein thecalibration parameters comprise at least one of (i) an adjustment to afield of view, (ii) an angular range setting, or (iii) a directionaladjustment.
 32. A vehicle comprising: a sensor system comprising asensor, the sensor system being calibrated based at least in part onfirst sensor data recorded while the senor is positioned on the moveablesurface within an indoor facility and one or more calibrationparameters; and a control system comprising one or more processors andone or more memory resources storing instructions that, when executed bythe one or more processors, cause the one or more processors to performoperations comprising: receiving second sensor data recorded by thesensor system while the vehicle operates in an outdoor environment; andproviding control commands to control motion of the vehicle based atleast in part on the second sensor data recorded by the sensor systemwhile the vehicle operates in the outdoor environment.
 33. The vehicleof claim 32, wherein the operations further comprise: receiving dataassociated with calibration of the sensor system of the vehicle; andgenerate a set of adjustment commands for the sensor system based atleast in part on the data associated with calibration of the sensorsystem.
 34. The vehicle of claim 33, wherein the set of adjustmentcommands are indicative of an adjustment to at least one of (i) a powerof the sensor, (ii) a field of view of the sensor, (iii) a sensitivityof the sensor, or (iv) an alignment of the sensor.
 35. The vehicle ofclaim 32, wherein the sensor comprises a LIDAR sensor, a radar sensor,or a camera sensor.
 36. The vehicle of claim 32, wherein the moveablesurface is a rotatable surface located within the indoor facility. 37.The vehicle of claim 32, wherein the first sensor data is associatedwith at least one of a plurality of targets positioned within the indoorfacility.
 38. The vehicle of claim 32, wherein the calibrationparameters are associated with a particular type of sensor.
 39. Acomputer-implemented method comprising: receiving sensor data recordedby a sensor system comprising a sensor, the sensor positioned on amoveable surface, wherein the sensor data is associated with at leastone of a plurality of targets and the sensor data is recorded while thesensor system is positioned on the moveable surface; determining one ormore calibration parameters to calibrate the sensor system based atleast in part on the sensor data; and communicating data for calibrationof the sensor system based at least in part on the sensor data recordedby the sensor system and the one or more calibration parameters.
 40. Thecomputer-implemented of claim 39, wherein the sensor system is locatedon an autonomous vehicle.